deps(cw): regen clients with codegen at 0.25.x #6439

Hweinstock · 2025-01-27T17:01:12Z

Problem

The two streaming clients committed to the repo are generated via different versions of smithy-typescript-codegen. Each version of codegen pins the generated client to use specific versions of @aws-sdk/* and @smithy/* packages. The root project also consumes different versions of these packages. Therefore, the project is currently consuming three different versions of many @aws-sdk/* and @smithy/* packages.

This is problematic because it can cause dependency conflicts. Certain versions of @aws-sdk/* and @smithy/* packages are only compatible with certain other versions. For more information on this, see the discussion involving help from an SDK team member explaining this problem.

Solution

Regenerate both clients with the same version of codegen 0.25.x.
Pin all necessary versions to ensure a single version of @aws-sdk/* and @smithy/* packages.
Add documentation to the build instructions that these clients must stay in-sync (in terms of codegen).
Add documentation regarding updating @aws-sdk/* packages.

Treat all work as PUBLIC. Private feature/x branches will not be squash-merged at release time.
Your code changes must meet the guidelines in CONTRIBUTING.md.
License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

github-actions · 2025-01-27T17:01:41Z

This pull request modifies code in src/* but no tests were added/updated.
- Confirm whether tests should be added or ensure the PR description explains why tests are not required.

trivikr · 2025-01-28T20:35:12Z

The [email protected] published <v3.696.0 of JS SDK v3 in aws/aws-sdk-js-v3#6673

Since this is a monorepo which has a single lock file across all clients, the version of client packages need to be updated to <3.696.0 in https://github.com/Hweinstock/aws-toolkit-vscode/blob/3c37c5777ac395c5f1b59da3d7599b1e554210d8/packages/core/package.json#L499-L504

Since AWS SDK for JavaScript v3 follows fixed versioning, the <3.696.0 version can be applicable to other @aws-sdk/* dependencies too.

trivikr · 2025-01-28T20:40:17Z

As there's another codegen client @amzn/amazon-q-developer-streaming-client, it should also be codegen with 0.25.x as it's also part of monorepo.

https://github.com/Hweinstock/aws-toolkit-vscode/blob/3c37c5777ac395c5f1b59da3d7599b1e554210d8/src.gen/%40amzn/amazon-q-developer-streaming-client/package.json#L23-L34

It currently uses old codegen version.

Hweinstock · 2025-01-29T15:26:05Z

I pinned the @aws-sdk/* packages to be <3.696.0 and regenerated the other client with codegen version 0.25.x. Are there any other steps I should take here? Otherwise planning to retry with 0.26.x once its available.

trivikr · 2025-01-29T16:52:01Z

I pinned the @aws-sdk/* packages to be <3.696.0

I see that you've use exact version 3.696.0.
https://github.com/Hweinstock/aws-toolkit-vscode/blob/b1470ed160aa114012e8cbee9dbac00969c9a8df/packages/core/package.json#L499-L510

Can you use <3.696.0 for all @aws-sdk/* dependencies?

There's no issue with using exact version as it'll be just one version above previous one.
But since all JS SDK clients follow fixed versioning, and we don't release new versions if there are no updates in specific clients or it's dependencies, we suggested <.

trivikr · 2025-01-29T16:57:46Z

For a detailed explanation, if you search for @aws-sdk/client-sso-oidc in package-lock.json, it contains both 3.693.0 and 3.696.0
https://github.com/Hweinstock/aws-toolkit-vscode/blob/b1470ed160aa114012e8cbee9dbac00969c9a8df/package-lock.json

It contains 3.693.0 since generated clients depend on it https://github.com/Hweinstock/aws-toolkit-vscode/blob/b1470ed160aa114012e8cbee9dbac00969c9a8df/src.gen/%40amzn/amazon-q-developer-streaming-client/package.json#L23
It contains 3.696.0 since core depends on it https://github.com/Hweinstock/aws-toolkit-vscode/blob/b1470ed160aa114012e8cbee9dbac00969c9a8df/packages/core/package.json#L504

If you use <3.696.0 on core package, it'll use 3.693.0.
Similarly if you pin other non-client @aws-sdk/* packages to <3.696.0, other dependency conflicts will get resolved.

Hweinstock · 2025-01-29T18:30:09Z

I see, thanks for the explanation that makes a lot of sense. I switched it to to be <3.696.0 and verified there is only one version (3.693.0) of @aws-sdk/client-sso-oidc in the package-lock now. Unfortunately, it doesn't seem to solve the smithy package conflicts, but this version pinning done here has eliminated all the errors I was originally seeing with type mismatches so I think we're getting closer.

trivikr · 2025-01-29T18:42:55Z

I'm glad that versioning issues with @aws-sdk/* are no longer there.

Do update your Contributor Guide with learnings from this PR. The AWS SDK for JavaScript team will take a backlog to write a doc on versioning from this experience - possibly in our Developer Guide. We'll get it reviewed from you internally before publishing.

trivikr · 2025-01-29T18:46:59Z

Tackling @smithy/* versioning is little tricky, but we use latest for those packages as they can be used outside of AWS too. Your monorepo can do the same.

I see that you're using 5 dependencies from @smithy

        "@smithy/middleware-retry": "^2.3.1",
        "@smithy/protocol-http": "^3.3.0",
        "@smithy/service-error-classification": "^2.1.5",
        "@smithy/shared-ini-file-loader": "^2.2.8",
        "@smithy/util-retry": "^2.2.0",

https://github.com/Hweinstock/aws-toolkit-vscode/blob/83a18cc84e7a4317048b4ddc12357c65ae310e3e/packages/core/package.json#L514-L518

An easy fix would be to copy the same version used by your code generated packages. i.e.

        "@smithy/middleware-retry": "^3.0.27",
        "@smithy/protocol-http": "^4.1.7",
        "@smithy/service-error-classification": "^3.0.11", // not direct client dependency, found from package-lock
        "@smithy/shared-ini-file-loader": "^3.1.12", // not direct client dependency, found from package-lock
        "@smithy/util-retry": "^3.0.10",

Since it's the latest major version, you can use ^N.0.0 if it simplifies your tooling

        "@smithy/middleware-retry": "^3.0.0",
        "@smithy/protocol-http": "^4.0.0",
        "@smithy/service-error-classification": "^3.0.0",
        "@smithy/shared-ini-file-loader": "^3.0.0",
        "@smithy/util-retry": "^3.0.0",

Hweinstock · 2025-01-30T14:55:02Z

Thanks for the help! I was able to get it building by fixing the smithy versions. The tests are now failing since the code that relied on the older versions of @smithy/* packages does not work on the newer versions, and I implemented some workarounds just so that it builds for now.

Assuming we can update that code to work with the newer smithy versions, I want to make sure I understand how to avoid this in the future. Here is my understanding of what we had to do to fix this:

generate all clients from the same version of codegen. (in this case 0.25.x).
fix @aws-sdk/* package versions in the root to be no-newer than the version fixed by codegen. (in this case <3.696.0)
fix @smithy/* package versions to the same major versions used by the generated clients (including indirect dependencies from the package-lock.json) on a per package basis. I emphasize the last point because its not intuitive that the generated client uses "@smithy/middleware-retry": "^3.0.27" and "@smithy/protocol-http": "^4.1.7" despite coming from the same codegen version, so one needs to check that each package lines up.

Feel free to let me know if I missed any important steps above. Also a few follow up questions:

Is there any recommended way to go about enforcing these constraints? Any solutions from other teams that have been effective?
Is this something we will always have to deal with when consuming generated typescript clients from smithy? Are there any plans to improve this experience?
Since we don't own the packages that generate these clients, is there anything we can do when their codegen versions inevitably diverge? Is our only option to block merges/changes to them in this repo until they are in sync?
How do we determine the version to fix based on the codegen version? Does it require manually looking at the generated package.json and taking the newest version of an @aws-sdk/* package?
What happens if we need to update a package we consume in the toolkit but there isn't a version of codegen available that lets us pin to a newer version?
Can you provide some insight into why this happens? I was under the impression that npm allowed packages to manage their own dependencies, so you could have different versions of the same package existing in the same repository, where each package used the version of dependencies it needed. However, if that were the case, this shouldn't be happening so what am I missing here?

Thank you again for your help on this issue! This has been a huge headache for us, and its refreshing to see a path out of it.

kuhe · 2025-02-12T16:57:28Z

generate all clients from the same version of codegen. (in this case 0.25.x).

yes

fix @aws-sdk/* package versions in the root to be no-newer than the version fixed by codegen. (in this case <3.696.0)

yes, but use the highest version you can see that was output by code generation. If code generation output contains:

@aws-sdk/core: "3.735.0",
@aws-sdk/some-package: "3.733.0",

use <=3.735.0 for everything.

fix @smithy/* package versions to the same major versions used by the generated clients (including indirect dependencies from the package-lock.json) on a per package basis. I emphasize the last point because its not intuitive that the generated client uses "@smithy/middleware-retry": "^3.0.27" and "@smithy/protocol-http": "^4.1.7" despite coming from the same codegen version, so one needs to check that each package lines up.

You shouldn't need to specify any @smithy/* package versions. Code generation will output e.g. ^5.x.y of some smithy package, and because you've pinned the @aws-sdk/client-* packages to the contemporaneous version, e.g. @aws-sdk/client-s3@<=3.735.0, the AWS SDK clients are using the same range of smithy dependency. This means there will only be one instance of any given core smithy or core aws-sdk package(s).

Is there any recommended way to go about enforcing these constraints? Any solutions from other teams that have been effective?

No

Is this something we will always have to deal with when consuming generated typescript clients from smithy? Are there any plans to improve this experience?

Yes. We may publish smithy-typescript maven versions with higher frequency in the future. We may create a dynamic client that accepts a model at runtime in the future, long term.

Since we don't own the packages that generate these clients, is there anything we can do when their codegen versions inevitably diverge? Is our only option to block merges/changes to them in this repo until they are in sync?

Sounds like yes.

How do we determine the version to fix based on the codegen version? Does it require manually looking at the generated package.json and taking the newest version of an @aws-sdk/* package?

Yes

What happens if we need to update a package we consume in the toolkit but there isn't a version of codegen available that lets us pin to a newer version?

use NPM overrides or request a newer release of the code generators.

Can you provide some insight into why this happens? I was under the impression that npm allowed packages to manage their own dependencies, so you could have different versions of the same package existing in the same repository, where each package used the version of dependencies it needed. However, if that were the case, this shouldn't be happening so what am I missing here?

NPM duplication resolves runtime conflicts of packages needing two different versions of some NPM package. This creates a class of runtime issues relating to inheritance checks but otherwise allows deconfliction.

TypeScript is a fictional overlay that is of no concern to the package duplication system. Type errors from duplication are not addressed by it.

… packages. (#6474) ## Problem The auth code relies on old versions of `@aws-sdk/*` that have since been deprecated or are no longer backward compatible, making versions bumps impossible. - `@aws-sdk/credential-provider-imds` has since been [deprecated](https://www.npmjs.com/package/@aws-sdk/credential-provider-imds) - `fromIni` from `@aws-sdk/credential-provider-ini` no longer supports passing a `loadedConfig`. - `AssumeRoleParams` is no longer exported by `@aws-sdk/credential-provider-ini`. We need to be able to bump these `@aws-sdk/*` package versions to continue to consume newer generated clients. Being pinned to older versions is also a security risk. See #6439 for more information. ## Solution - write custom credentials provider to replace `fromIni` with `loadedConfig` option. - drop dependency on `@aws-sdk/credential-provider-ini` since its no longer used. - add direct dependency on `@aws-sdk/credential-provider-env` since this was installed as part of `@aws-sdk-credential-provider-ini` before. - Fix many (not all) of the deprecation warnings in auth code related to credentials provider. ### Custom Credentials Provider Before, we used `fromIni` with the `loadedConfig` option which allows us to avoid reading the config file from disk on each credentials fetch and allows us to merge the current credentials with those found in the `.ini` file. To achieve the same behavior without the `loadedConfig` option, we need to write our own credentials provider that supports MFA and role assumption, and returns the desired merged credentials, rather than reading from disk. ### Testing - Manually verify this role assumption works by following the steps [here](https://docs.aws.amazon.com/sdkref/latest/guide/access-assume-role.html). - Manually verify MFA works via adapting [this](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-role.html#:~:text=This%20policy%20allows%20the%20user,they%20authenticate%20by%20using%20MFA.&text=Next%2C%20add%20a%20line%20to,by%20the%20role's%20trust%20policy.&text=The%20mfa_serial%20setting%20can%20take,command%20with%20this%20profile%20fails.&text=The%20second%20profile%20entry%2C%20role,%22:%20%5B%20%7B%20...). (Used DuoMobile) - Add unit tests with API calls stubbed. ## Future Work - There are two tests that can now be re-enabled because of this version bump, undoing db27ebb - The steps to test role assumption could become an integ/e2e test. Right now requires setting many resources up in console, but perhaps this can all be done by the SDKs with an account on admin access. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

jpinkney-aws

Add documentation to the build instructions that these clients must stay in-sync (in terms of codegen).
Add documentation regarding updating @aws-sdk/* packages.

Is that internal documentation? I'm guessing the generator update instructions?

Hweinstock · 2025-02-13T21:30:38Z

Yes, its internal. Could be worth summarizing it and adding it into the public facing docs at some point. This problem mostly affects developers trying to regenerate the streaming clients, so I think the public use-case is minimal.

leigaol · 2025-02-14T18:42:23Z

The vsix from the package test is working fine in my VS Code after installation.

… packages. (aws#6474) The auth code relies on old versions of `@aws-sdk/*` that have since been deprecated or are no longer backward compatible, making versions bumps impossible. - `@aws-sdk/credential-provider-imds` has since been [deprecated](https://www.npmjs.com/package/@aws-sdk/credential-provider-imds) - `fromIni` from `@aws-sdk/credential-provider-ini` no longer supports passing a `loadedConfig`. - `AssumeRoleParams` is no longer exported by `@aws-sdk/credential-provider-ini`. We need to be able to bump these `@aws-sdk/*` package versions to continue to consume newer generated clients. Being pinned to older versions is also a security risk. See aws#6439 for more information. - write custom credentials provider to replace `fromIni` with `loadedConfig` option. - drop dependency on `@aws-sdk/credential-provider-ini` since its no longer used. - add direct dependency on `@aws-sdk/credential-provider-env` since this was installed as part of `@aws-sdk-credential-provider-ini` before. - Fix many (not all) of the deprecation warnings in auth code related to credentials provider. Before, we used `fromIni` with the `loadedConfig` option which allows us to avoid reading the config file from disk on each credentials fetch and allows us to merge the current credentials with those found in the `.ini` file. To achieve the same behavior without the `loadedConfig` option, we need to write our own credentials provider that supports MFA and role assumption, and returns the desired merged credentials, rather than reading from disk. - Manually verify this role assumption works by following the steps [here](https://docs.aws.amazon.com/sdkref/latest/guide/access-assume-role.html). - Manually verify MFA works via adapting [this](https://docs.aws.amazon.com/cli/v1/userguide/cli-configure-role.html#:~:text=This%20policy%20allows%20the%20user,they%20authenticate%20by%20using%20MFA.&text=Next%2C%20add%20a%20line%20to,by%20the%20role's%20trust%20policy.&text=The%20mfa_serial%20setting%20can%20take,command%20with%20this%20profile%20fails.&text=The%20second%20profile%20entry%2C%20role,%22:%20%5B%20%7B%20...). (Used DuoMobile) - Add unit tests with API calls stubbed. - There are two tests that can now be re-enabled because of this version bump, undoing aws@db27ebb - The steps to test role assumption could become an integ/e2e test. Right now requires setting many resources up in console, but perhaps this can all be done by the SDKs with an account on admin access. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

The two streaming clients committed to the repo are generated via different versions of `smithy-typescript-codegen`. Each version of codegen pins the generated client to use specific versions of `@aws-sdk/*` and `@smithy/*` packages. The root project also consumes different versions of these packages. Therefore, the project is currently consuming three different versions of many `@aws-sdk/*` and `@smithy/*` packages. This is problematic because it can cause dependency conflicts. Certain versions of `@aws-sdk/*` and `@smithy/*` packages are only compatible with certain other versions. For more information on this, see the discussion involving help from an SDK team member explaining this problem. - Regenerate both clients with the same version of codegen `0.25.x`. - Pin all necessary versions to ensure a single version of `@aws-sdk/*` and `@smithy/*` packages. - Add documentation to the build instructions that these clients must stay in-sync (in terms of codegen). - Add documentation regarding updating `@aws-sdk/*` packages. --- - Treat all work as PUBLIC. Private `feature/x` branches will not be squash-merged at release time. - Your code changes must meet the guidelines in [CONTRIBUTING.md](https://github.com/aws/aws-toolkit-vscode/blob/master/CONTRIBUTING.md#guidelines). - License: I confirm that my contribution is made under the terms of the Apache 2.0 license.

deps: add in generated client

b6a853e

fix: add missing type annotation

3c37c57

Hweinstock added 4 commits January 29, 2025 09:56

fix: pin client versions used by toolkit

a3c9adf

deps: regen other client to fix to same version

7450af0

deps: fix other @aws-sdk/* packages

7a33dcc

deps: update package-lock

b1470ed

Hweinstock changed the title ~~deps(cw): regen cw client with codegen at 0.25.x (IGNORE)~~ deps(cw): regen clients with codegen at 0.25.x (IGNORE) Jan 29, 2025

deps: avoid fixing to specific version

83a18cc

Hweinstock added 3 commits January 29, 2025 14:06

deps: pin smithy major versions

b3e02c9

deps: add back missing module

046c3fb

refactor: gross workarounds for it to build

8b44fdd

Hweinstock added 5 commits January 30, 2025 13:00

refactor: implement custom credentials provider

548fb51

refactor: rename profile field to match expected

96c7759

docs: add documentation link in implementation

a4a2709

deps: remove deprecated dependencies

49195c3

refactor: add stronger typing

d2f3c4f

Hweinstock mentioned this pull request Jan 30, 2025

deps(auth): remove dependence on deprecated and outdated @aws-sdk/* packages. #6474

Merged

Hweinstock added 3 commits January 31, 2025 11:34

docs: remove documentation link since its already in the types

d7bede6

test: implement tests for behavior of credentials provider

d7a9ca3

merge: resolve package.json conflicts

cee9f55

Hweinstock added 2 commits February 3, 2025 10:53

deps: remove dependence on ini and add env

c5aa07f

merge: resolve package-lock conflicts

bf3af8f

This was referenced Feb 10, 2025

deps(clients): regen qdev streaming client (IGNORE) #6335

Closed

deps(client): regen cw streaming client (ignore) #6345

Closed

Hweinstock added 7 commits February 11, 2025 09:53

refactor: simplify conditional args

20552b8

docs: add section with links for shared credentials file

21b8b31

fix: avoid importing from index.ts

4befa8c

Merge branch 'master' into auth/loadedConfigFile

c3ac3d8

Merge branch 'master' into auth/loadedConfigFile

7a58a30

merge: fix conflicts with upstream

0242b59

merge: bring in auth changes

a54d8c0

Hweinstock changed the title ~~deps(cw): regen clients with codegen at 0.25.x (IGNORE)~~ deps(cw): regen clients with codegen at 0.25.x Feb 13, 2025

Hweinstock added 2 commits February 13, 2025 15:43

merge: resolve conflicts

4a847f5

refactor: undo unnecessary import change

3039bdb

Hweinstock marked this pull request as ready for review February 13, 2025 21:01

Hweinstock requested review from a team as code owners February 13, 2025 21:01

jpinkney-aws approved these changes Feb 13, 2025

View reviewed changes

justinmk3 approved these changes Feb 13, 2025

View reviewed changes

nkomonen-amazon approved these changes Feb 13, 2025

View reviewed changes

leigaol approved these changes Feb 14, 2025

View reviewed changes

Hweinstock merged commit 4fe5168 into aws:master Feb 14, 2025
25 of 26 checks passed

Hweinstock mentioned this pull request Feb 18, 2025

deps: bump the aws-sdk group across 1 directory with 13 updates #6608

Open

Hweinstock deleted the regenClient/0.25.x branch February 18, 2025 18:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

deps(cw): regen clients with codegen at 0.25.x #6439

deps(cw): regen clients with codegen at 0.25.x #6439

Hweinstock commented Jan 27, 2025 •

edited

Loading

github-actions bot commented Jan 27, 2025

trivikr commented Jan 28, 2025 •

edited

Loading

trivikr commented Jan 28, 2025 •

edited

Loading

Hweinstock commented Jan 29, 2025

trivikr commented Jan 29, 2025

trivikr commented Jan 29, 2025

Hweinstock commented Jan 29, 2025

trivikr commented Jan 29, 2025

trivikr commented Jan 29, 2025 •

edited

Loading

Hweinstock commented Jan 30, 2025 •

edited

Loading

kuhe commented Feb 12, 2025

jpinkney-aws left a comment

Hweinstock commented Feb 13, 2025

leigaol commented Feb 14, 2025

deps(cw): regen clients with codegen at 0.25.x #6439

deps(cw): regen clients with codegen at 0.25.x #6439

Conversation

Hweinstock commented Jan 27, 2025 • edited Loading

Problem

Solution

github-actions bot commented Jan 27, 2025

trivikr commented Jan 28, 2025 • edited Loading

trivikr commented Jan 28, 2025 • edited Loading

Hweinstock commented Jan 29, 2025

trivikr commented Jan 29, 2025

trivikr commented Jan 29, 2025

Hweinstock commented Jan 29, 2025

trivikr commented Jan 29, 2025

trivikr commented Jan 29, 2025 • edited Loading

Hweinstock commented Jan 30, 2025 • edited Loading

kuhe commented Feb 12, 2025

jpinkney-aws left a comment

Choose a reason for hiding this comment

Hweinstock commented Feb 13, 2025

leigaol commented Feb 14, 2025

Hweinstock commented Jan 27, 2025 •

edited

Loading

trivikr commented Jan 28, 2025 •

edited

Loading

trivikr commented Jan 28, 2025 •

edited

Loading

trivikr commented Jan 29, 2025 •

edited

Loading

Hweinstock commented Jan 30, 2025 •

edited

Loading